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SECTION  1 


INTRODUCTION 


This  report  considers  ways  tc  evaluate  the  design  of  software 
supporting  the  user-system  interface  (USI)  to  computer-based 
information  systems . 

Section  1  discusses  USI  software  generally,  and  the  <.urrent 
MITRE  development  of  guidelines  for  USI  software  design,  sponsored 
by  the  Air  Force  Electronic  Systems  Division  (ESD) .  Alternative 
apnroaches  to  USJ  software  evaluation  are  considered,  including  the 
possibility  of  evaluating  software  by  comparison  with  design 
guidelines . 

Section  2  proposes  a  desigxi  evaluation  checklist  derived  from 
published  guidelines.  The  checklist  .’s  formatted  to  permit 
differential  weighting  and  ra'ir.g  of  various  itemized  design 
features.  The  checklist  itsc'r  is  presented  in  Appendix  A,  and  a 
sample  application  of  the  checklist  is  illustrated  In  Appendix  B. 

Section  3  describes  potential  use  of  the  design  evaluation 
checklist  if  it  were  implemented  as  an  on-line  computer  tool,  with 
aids  to  facilitate  review  and  tailoring  of  guidelines  material,  and 
to  support,  the  entry,  analysis,  and  reporting  of  USI  software  design 
evaluation.  A  sample  design  analysis  that  might  be  generated  by 
such  a  computer  tool  is  illustrated  in  Appendix  C. 


USI  SOFTWARE 

What  is  the  user-system  interface?  Here  the  phrase  "user- 
system  interface"  is  defined  broadly  to  include  all  aspects  of 
design  that  affect  a  system  user's  participation  in  data  handling 
transactions.  The  word  "user"  is  intended  to  include  managers  and 
maintainers  of  a  system  as  well  as  its  operators. 

Directly  observable  aspects  of  the  user  interface  include  the 
physical  work  environment  and  the  hardware  facilities  at  a  user's 
work  station.  Such  physical  features,  sometimes  called  the  man- 
machine  interface,  have  been  the  subject  of  conventional  human 
engineering  study.  There  the  concern  is  for  illumination,  seating, 
workplace  arrangement,  keyboard  layout,  display  contrast,  symbol 
size,  etc. 
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Good  design  of  the  physical  workplace  is  important,  of  course, 
but  by  itself  is  not  sufficient  to  ensure  effective  job  performance. 
Also  important  are  less  tangible  aspects  of  information  system 
design  --  the  ways  in  which  data  are  stored  and  manipulated, 
including  paper  files  and  forms,  if  any,  and  the  procedures  and 
processing  logic  that  govern  data  handling  transactions. 

Forms,  procedures,  and  logic  involve  software  design,  the 
design  of  computer  programs  to  permit  hardware  and  paper  to  be  used 
in  conjunction  with  automated  data  processing.  Effective  design  of 
USI  software  is  critical  to  system  performance. 

In  a  recent  survey,  people  involved  in  design  of  information 
systems  were  asked  to  estimate  the  percent  of  operational  software 
devoted  to  implementing  the  user  interface.  Overall,  the  average 
estimate  was  that  USI  design  comprises  30-35  percent  of  operational 
software.  Estimates  for  individual  systems  ranged  from  3  to  100 
percent  (Smith  and  Mosier,  1984a). 

Because  USI  software  is  important,  and  also  expensive,  we  must 
find  ways  to  improve  its  design.  For  system  designers,  guidelines 
have  been  proposed  to  help  define  requirements  for  the  design  of 
user  interface  software.  For  system  users,  and  for  their  agents 
such  as  ESD  who  sponsor  system  development,  we  need  corresponding 
tools  for  evaluating  USI  software  design. 


USI  DESIGN  GUIDELINES 

Over  the  past  several  years,  there  has  been  a  continuing  effort 
at  ESD/MITRE,  under  Project  522A,  to  compile  guidelines  for  the 
design  of  user  interface  software.  In  that  effort,  we  have  tried  to 
distill  the  current  wisdom  of  USI  design  experts  in  order  to  provide 
comprehensive  advice  to  designers. 

Each  year  our  published  compilation  of  guidelines  has  grown 
larger.  The  most  recent  report  on  this  subject  proposes  some  679 
guidelines  in  450  pages,  covering  the  general  functional  areas  of 
data  entry,  data  display,  sequence  control,  user  guidance,  data 
transmission,  and  data  protection  (Smith  and  Mosier,  1984b). 

Those  guidelines  will  undergo  further  review  and  revision.  An 
improved  version  of  the  USI  design  guidelines  will  be  published  in 
1985.  Meanwhile,  the  current  report  can  provide  a  good  starting 
point  for  deriving  specific  rules  to  govern  user  interface  design  in 
any  particular  system  development  program. 
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To  help  assure  broad  applicability  USI  design  guidelines  are 
expressed  in  general  terms,  which  may  then  be  amplified  by  specific 
commentary  and  examples.  A  sample  of  general  guidelines  pertaining 
to  data  entry  functions  is  presented  in  Appendix  A. 

Topical  organization  of  the  guidelines  corresponds  to  defined 
functional  USI  capabilities.  Thus  it  may  be  possible  in  some  degree 
to  tailor  guidelines  to  USI  requirements.  If  a  function  is  not 
required  for  a  particular  system,  then  design  guidelines  related  tc 
that  function  will  not  be  applicable. 

In  many  instances,  software  designers  must  interpret  the 
guidelines  based  on  task  analysis,  anticipated  data  processing 
capabilities  and  user  skills,  in  order  to  meet  the  needs  of  a 
particular  system.  In  effect,  designers  must  translate  or  convert 
guidelines  into  specific  design  rules. 

The  establishment  of  rules  for  user  interface  design  will  be  of 
significant  value  in  system  development,  and  should  be  specifically 
included  in  a  design  contractor's  statement  of  work.  The  contractor 
should  be  required  to  establish  design  rules  in  advance  of  USI 
software  implementation.  Contractor  documentation  of  USI  design 
rules,  and  any  subsequent  revisions,  should  be  maintained  thereafter 
on  a  current  basis  to  aid  design  coordination  throughout  system 
development  (Smith,  1984). 

It  must  be  recognized  that  establishment  of  agreed  rules  will 
not  in  itself  assure  effective  user  interface  design.  No  matter  how 
clearly  they  are  stated,  those  rules  will  still  be  subject  to 
varying  interpretation  by  software  designers.  Moreover,  it  is 
inevitable  that  some  design  trade-offs  will  be  required  in  the 
practical  application  of  different  rules. 

But  the  process  of  establishing  and  documenting  the  rules 
should  ensure  that  serious  consideration  is  given  to  this  aspect  of 
software  design.  And  the  existence  of  agreed  rules  should  help 
focus  attention  and  clarify  discussion  of  user  interface  design 
issues  among  the  sponsors,  designers  and  ultimate  users  of  a  system. 


USI  DESIGN  EVALUATION 

Design  guidelines  provide  a  tool  that  can  be  applied  Hy  system 
contractors,  i.e.,  by  people  responsible  for  conceiving  and 
implementing  USI  design.  Some  corresponding  tool  should  be  devised 
for  people  who  are  responsible  for  evaluating  USI  design.  Those 
people  nay  be  the  system  users,  or  they  may  represent  contracting 
agencies  such  as  ESD  who  act  on  behalf  of  the  eventual  system  users. 
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Effective  design  evaluation  requires  emphasis  throughout  the 
course  of  system  development.  Evaluation  must  begin  with  timely 
review  of  design  proposals  in  advance  of  implementation,  continue 
with  review  and  assessment  of  design  prototypes,  through  operational 
testing  of  the  fully  designed  system. 

Alternative  Approaches 

There  are  several  possible  approaches  to  design  evaluation, 
including  analytic  modeling,  performance  testing,  comparison  with 
design  guidelines,  review  of  specifications  and  design  proposals, 
user  surveys,  etc.  All  of  these  approaches  can  play  a  useful  role. 
The  first  three  deserve  further  comment. 

Analysis  and  Modeling 

Where  we  seek  insight  into  the  processes  that  underlie 
information  handling  tasks,  detailed  analysis  of  the  user  actions 
that  are  required  to  perform  various  tasks  permits  the  creation  of 
models  to  predict  the  effects  of  system  design  on  operational 
performance.  Task  modeling  can  help  assess  design  differences  among 
different  systems,  or  to  predict  the  effects  of  design  changes  to  a 
particular  system.  Accuracy  of  prediction  will  depend  upon  the 
validity  of  the  models,  which  is  subject  to  experimental  testing  in 
the  laboratory  and  operational  testing  ir.  actual  system  use.  The 
results  of  such  analysis  and  modeling  must  be  translated  into 
specific  design  recommendations,  which  will  require  considerable 
judgment . 

Performance  Testing 


In  applications  where  operational  performance  is  critical,  the 
most  direct  means  of  evaluation  is  to  test  the  performance  of  system 
users  on  various  characteristic  ("benchmark")  information  handling 
tasks.  A  range  of  users  and  tasks  must  be  tested  to  represent 
fairly  the  actual  system.  Performance  testing  must  wait  for 
completion  of  system  design  (or  at  least  a  working  prototype),  or 
else  must  be  based  on  detailed  modeling  if  done  in  advance  of  system 
design.  Such  testing  requires  a  considerable  investment  of  time  and 
effort.  The  results  of  controlled  testing  should  provide  a  gcod 
prediction  of  actual  system  performance,  but  will  require  careful 
analysis  and  judgment  in  order  to  specify  needed  design 
improvements . 

Comparison  with  Design  Guidelines 

Where  specific  design  recommendations  are  needed,  the  quickest 
way  to  evaluate  the  user  interface  may  be  to  examine  design  features 
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provided  in  a  system  (or  proposed  features  for  a  system  still  under 
development)  and  compare  those  features  with  established  design 
guidelines.  Establishment  of  agreed  guidelines  will  require  the 
collective  judgment  of  the  sponsors  (procuring  agency,  marketing 
department,  etc.),  designers,  and  prospective  users  of  the  system. 
Further  judgment  will  be  needed  to  assess  design  compliance  with 
agreed  guidelines.  The  validity  of  this  approach  to  design 
evaluation  will  depend  upon  the  good  judgment  of  the  people  involved 
in  compiling  and  applying  the  guidelines.  Validity  might  be 
measured  by  performance  testing,  although  that  process  will  prove 
impractical  as  the  number  of  design  features  considered  (and  their 
possible  interactions)  grows  very  large. 

Comparative  Advantages 

These  three  approaches  to  design  evaluation  are  complementary. 

A  program  of  analysis  and  modeling  will  require  performance  testing 
for  its  validation.  Performance  testing  must  in  turn  be  analyzed  to 
interpret  its  results.  And  both  approaches  will  benefit  from 
reference  to  design  guidelines  in  order  to  generate  specific  design 
recommendations.  For  effective  system  development  and  evaluation, 
all  three  approaches  may  be  needed.  The  comparative  strengths  and 
weaknesses  of  different  approaches  are  summarized  here: 


Approaches 

to  Evaluation 

Goals  of  Evaluation 

Analysis , 
Modeling 

Performance 

Testing 

Design 

Guidelines 

Insight  into  task 
processes 

Good 

Fair 

Poor 

Performance 

prediction 

Fair 

Good 

Poor 

Specific  design 
r  ecommendat  ions 

Fair 

Fair 

Good 

Ease  and  speed 
of  evaluation 

Fair 

Poor 

Good 

Evaluation  early  in 
system  development 

Fair 

Poor 

Good 

User  involvement  in 
design  decisions 

Poor 

Fair 

Good 
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Problems  of  Software  Evaluation 


System  evaluation  must  cover  both  hardware  and  software  design. 
As  a  practical  matter,  the  evaluation  of  proposed  hardware  may  be 
relatively  straightforward..  User  interface  hardware  can  be  checked 
for  compliance  with  the  provisions  of  current  DoD  human  engineering 
standards  for  equipment  design,  MIL-STD-1472,  as  well  as  with  system 
specifications.  Some  aspects  of  hardware  design  can  be  assessed 
simply  by  observation.  Are  the  corners  of  equipment  enclosures 
rounded  rather  than  sharp?  Just  look  at  the  blueprints,  or 
catalogs,  or  actual  equipment,  and  see. 

By  contrast,  evaluating  user  interface  software  will  be  more 
difficult,  particularly  in  advance  of  design  implementation.  For 
many  ESD  system  acquisitions,  it  is  not  possible  to  buy  off-the- 
shelf  software,  out  of  a  catalog.  Instead,  user  interface  software 
must  often  be  designed  anew  for  each  system  application.  And  the 
"blueprints"  for  a  proposed  new  software  design  may  be  hard  to  read 
and  evaluate. 

Evaluation  of  USI  software  may  pose  difficult  problems  even 
after  its  implementation.  It  may  be  necessary  to  operate  the  system 
under  controlled  conditions  to  confirm  that  design  objectives  have 
been  achieved. 

Are  data  displays  formatted  consistently  from  one  transaction 
to  another?  We  may  need  to  generate  different  displays  and  compare 
them.  To  facilitate  that  comparison,  we  may  need  to  request 
hard-copy  printouts  of  various  displays. 

Are  error  messages  worded  clearly  and  consistently?  That  may 
be  harder  to  assess.  We  cannot  expect  an  evaluator  to  make  every 
conceivable  type  of  error  while  operating  a  system,  trying  to 
generate  a  full  range  of  error  messages.  Instead,  we  may  need  to 
examine  a  detailed  software  design  specification  in  which  all  error 
messages  have  been  spelled  out. 

What  other  features  should  we  examine  in  evaluating  USI 
software  design?  The  range' will  extend  far  beyond  considerations  of 
display  formatting  and  the  wording  of  error  messages.  We  need  some 
comprehensive  listing  of  all  significant  features  that  should  be 
considered. 

Specificity  is  important  here.  For  design  evaluation  to  be 
useful,  it  must  produce  detailed  recommendations,  outlining  for 
system  designers  exactly  which  features  need  improvement.  It  is  not 
enough  to  shake  our  heads  and  say  that  a  user  interface  looks  clumsy 
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or  confusing,  that  a  system  is  difficult  to  learn  or  difficult  to 
use.  We  must  try  to  provide  a  more  specific;  diagnosis  of  design 
deficiencies . 

Tools  for  Software  Evaluation 

What  tools  can  we  devise  for  evaluating  USI  software  design? 

It  would  seem  sensible  to  start  with  the  published  design 
guidelines.  But  there  the  sheer  bulk  of  the  published  guidelines 
material  poses  a  serious  problem. 

We  cannot  expect  an  evaluator  to  read  through  a  book  containing 
hundreds  of  guidelines,  checking  each  one  for  design  compliance  and 
annotating  design  discrepancies  in  the  margins.  An  evaluator  will 
need  to  read  the  guidelines  once,  during  initial  assimilation  of  the 
material,  h  it  surely  woulc  i.ot  wish  to  do  so  repetitively  when 
assessing  the  user  interface  for  different  systems,  or  for  one 
system  at  different  stages. 

If  a  design  evaluation  were  actually  made  by  checking  and 
annotating  a  large  book  of  guidelines,  there  would  still  remain  the 
difficult  task  of  pulling  together  those  notes  to  create  a  coherent 
overall  assessment  of  the  strengths  and  weaknesses  of  an  evaluated 
user  interface  design. 

Certainly  the  published  guidelines  can  serve  as  a  starting 
point  for  USI  design  evaluation.  But  it  is  apparent  that  some  more 
compact  format  will  be  needed  for  recording  and  reporting  the 
results  of  design  evaluation.  In  the  next  section,  a  checklist 
format  is  proposed  for  that  purpose. 
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SECTION  2 

CHECKLIST  FOR  DESIGN  EVALUATION 


In  order  to  create  a  compact  design  evaluation  record,  we  must 
convert  the  fully-worded  guidelines  into  a  more  condensed  listing. 

In  such  a  list,  each  itemized  design  feature  could  be  checked  for 
compliance  and  discrepancies  noted.  The  resulting  checklist  format 
should  prove  relatively  easy  to  use  --  for  the  initial  recording  of 
evaluative  judgments,  and  for  the  subsequent  analysis  and  reporting 
of  design  evaluation.  That  checklist  is  presented  in  Appendix  A. 

The  use  of  checklists  for  design  evaluation  is,  of  course,  not 
a  new  idea.  A  recent  AMRL  report  recommends  checklists  among  other 
means  of  assessing  compliance  with  human  engineering  standards 
(Geer,  1981,  pp.  156-159).  Most  current  checklists  are  intended  to 
evaluate  hardware  rather  than  software,  but  checklists  have  also 
been  proposed  for  software  evaluation  (Cordes,  1980).  The  general 
approach,  in  terms  of  checklist  formatting  and  use,  should  prove 
much  the  same  for  evaluating  US I  software  design. 

The  checklist  presented  here  is  based  on  our  most  recent 
compilation  of  design  guidelines  for  USI  software  (Smith  and  Mosier, 
1984b).  Each  of  the  679  guidelines  is  represented  in  the  checklist 
by  its  title.  For  a  person  familiar  with  the  guidelines,  those 
titles  will  serve  as  reminders  of  the  guidelines  material.  Fi^st- 
time  checklist  users,  however,  will  need  to  refer  frequently  to  the 
source  guidelines  in  order  to  determine  exactly  what  design 
requirements  are  specified  there. 

The  checklist  proposed  here  is  simple  in  concept.  Each  design 
guideline  is  represented  as  an  item  entry  in  the  list.  After  each 
item  is  a  space  to  indicate  an  (optional)  weighting  of  importance. 
Next  there  is  space  to  record  an  estimate  of  compliance.  Finally 
there  is  space  in  which  to  note  whether  comments  have  been  added  to 
cite  specific  design  deficiencies  or  to  provide  other  kinds  of 
information. 

A  sample  showing  how  the  checklist  might  be  used,  with 
imaginary  entries  for  purposes  of  illustration,  is  shown  in  Appendix 
B.  That  sample  shows  just  a  portion  of  a  complete  USI  design 
evaluation  checklist,  the  portion  that  corresponds  with  guidelines 
for  data  entry  functions. 

Several  aspects  of  checklist  use  --  functional  coverage, 
weighting  importance,  rating  compliance,  and  annotation  —  all 
deserve  further  discussion. 
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FUNCTIONAL  COVERAGE 


A  user  of  the  checklist  will  note  significant  differences  in 
coverage  for  different  USI  functions-  Some  functions  are  thoroughly 
covered,  but  some  are  not.  Under  data  entry,  for  example,  there  are 
24  items  dealing  with  position  designation,  but  only  two  for 
direction  designation.  Graphic  functions  are  scarcely  covered  at 
all  in  the  present  checklist. 

This  differential  coverage  results,  of  course,  from  differences 
in  coverage  within  the  source  guidelines  material  from  which  the 
checklist  was  derived.  In  the  compilation  of  design  guidelines 
there  are  some  areas  in  which  our  knowledge  is  greater  than  in 
others,  and  thus  more  information  can  be  offered  to  interface 
designers.  This  condition  of  differential  knowledge  will  persist 
for  some  years  to  come,  and  so  the  problem  of  differential  coverage 
in  the  design  evaluation  checklist  will  persist  also. 

One  consequence  is  that  design  evaluation  may  be  biased  toward 
those  functions  that  are  covered  well  in  the  checklist,  and 
important  functions  with  little  coverage  may  be  neglected.  How 
great  is  this  problem?  Until  we  have  gained  experience  in  using  the 
design  evaluation  checklist,  we  cannot  answer  that  question. 

How  can  a  person  responsible  for  design  evaluation  cope  with 
this  problem?  Several  corrective  measures  are  available.  The 
evaluator  might  propose  additional  items  for  checklist  functions 
where  coverage  seems  inadequate,  thus  strengthening  weak  portions  of 
the  evaluation  tool.  For  example,  if  a  particular  system  has  an 
extensive  need  for  graphic  interaction,  then  it  may  be  possible  to 
establish  agreed  design  rules  for  graphic  functions  and  represent 
those  rules  by  items  added  to  the  present  checklist. 

As  an  alternative  or  supplementary  approach,  an  evaluator  might 
assign  exceptionally  high  weights  to  a  few  listed  items,  where  only 
those  few  deal  with  an  important  user  interface  function.  The 
process  of  differential  weighting  of  items  is  discussed  later  in 
this  section. 

Another  problem  of  checklist  coverage  is  specificity.  If 
design  guidelines  are  to  be  applicable  to  a  variety  of  system 
applications,  they  must  be  stated  generally.  This  will  also  be  true 
of  a  checklist  derived  from  those  guidelines. 

For  any  particular  system  rpplica^ion,  if  will  often  oe 
necessary  to  tailor  the  general  guidelines  to  become  system-specific 
design  rules.  It  is  only  such  specific  rules  that  will  provide 
eifective  guidance  to  designers. 
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Where  guidelines  are  revised  to  achieve  specificity,  then  it 
may  be  necessary  to  revise  corresponding  checklist  items  as  well. 
Perhaps  the  titles  of  checklist  items  need  not  be  changed,  but  those 
titles  should  be  interpreted  as  referring  to  whatever  system- 
specific  design  rules  have  been  adopted,  rather  than  to  the  general 
guidelines  that  underlie  those  rules. 


WEIGHTING  IMPORTANCE 

In  the  design  evaluation  checklist  there  will  be  hundreds  of 
items  to  be  considered.  For  any  particular  system  design,  some  of 
those  items  may  not  be  relevant.  For  example,  if  a  decision  is  made 
not  to  provide  multicolor  displays,  then  any  guidelines  pertaining 
to  color  coding  will  not  be  applicable  to  that  system. 

There  are  two  ways  to  deal  with  this  situation.  One  way  would 
be  to  tailor  the  design  evaluation  checklist  so  that  it  includes 
only  those  items  relevant  to  a  particular  system.  That  tailoring 
could  be  difficult  for  a  paper  checklist,  where  many  pages  might 
have  to  be  retyped.  Moreover,  it  might  prove  disconcerting  for  an 
experienced  evaluator  to  find  the  contents  of  the  checklist  changing 
for  each  different  system  evaluation. 

A  simpler  solution  is  to  preserve  the  checklist  in  standard 
format,  consistent  in  contents  and  ordering,  from  one  system  to 
another.  An  experienced  evaluator  will  find  each  item  where  it  is 
expected,  always  in  the  same  place.  Items  not  applicable  to  a 
particular  system  might  be  marked  "N/A"  in  the  course  of  desigi, 
evaluation,  and  could  be  ignored  as  long  as  that  designation  is 
retained. 

Just  which  items  are  not  applicable  can  presumably  be  decided 
by  users  (and/or  sponsors)  in  negotiation  with  the  system  design 
contractor,  or  perhaps  with  a  USI  software  subcontractor,  in  order 
to  help  define  requirements  before  USI  software  is  designed.  And 
that  decision  might  be  reviewed  periodically  during  successive, 
stages  of  system  design,  along  with  more  general  review  of  required 
USI  functional  capabilities.  -' 

Eliminating  from  consideration  any  checklist  items  that  have 
been  judged  not  applicable  to  a  particular  system  design,  one  may 
question  whether  the  remaining  items  are  all  equally  important.  An 
evaluator  tsighi  well  conclude  that  some  o£  those  items  are  more 
important  than  others,  and  wish  to  assign  differential  weightings  tc 
discriminate  among  the  various  items. 
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Such  we ign tings  covld  bo  simple  categorical  judgments  among 
ju:«t  a  fev  designations  --  perhaps  "Required"  or  "Desirable"  in 
addition  to  "Hot  Applicable’1.  Or  weighting  judgments  could  be 
represented  along  a  numeric  reals  --  perhaps  zero  to  ten,  where  zero 
denotes  items  judged  cu  be  not  applicable,  and  ten  denotes  iteirs 
judged  to  be  of  critical  importance.  (Such  numeric,  weightings  are 
illustrated  by  the  sample  in  Appendix  B.)  Numeric,  weightings  would 
offer  advantages  for  subsequent  computational  analysis  of  user 
interface  design  evaluation ,  as  discussed  in  Section  3. 

In  practice,  the  assignment  of  differential  weightings  to 
checklist  items  must  necessarily  be  judgmental.  For  any  particular 
item,  different  weightings  might  be  assigned  by  system  isers,  by 
software  designers,  or  by  their  human  engineering  consultants. 

The  checklist  can  help  focus  attention  on  design  decisions,  but 
cannot  tell  people  how  to  make  those  decisions.  Design  biases  may 
be  introduced  in  the  weighting  process.  Users  might  weight  heavily 
those  design  features  that  have  been  missing  in  the  past,  hoping 
thereby  to  encourage  their  inclusion  in  a  new  system.  Designers 
might  weight  heavily  those  features  that  they  believe  will  be  easy 
to  provide. 

Whose  judgment  should  prevail?  The  best  solution  is  to  ensure 
that  weightings  will  not  be  established  by  only  one  person.  All 
groups  concerned  with  system  development  should  have  an  opportunity 
to  participate  in  the  weighting  process. 


RATING  COMPLIANCE 

In  uuing  the  checklist,  an  evaluator  must  judge  for  each  item 
the  degree  of  compliance  in  a  proposed  or  implemented  user  interface 
design.  Different  rating  schemes  might  be  adopted  to  record  the 
evaluator's  judgment. 

One  might  adopt  a  simple  YES/NO  racing  to  indicate  judged 
design  compliance  for  each  checklist  item.  At  least  that  seems 
simple.  But  the  evaluator  will  SLill  have  to  exercise  judgment  as 
to  whether  the  observed  degree  of  design  compliance  is  substantial 
enough  to  justify  a  YES  rating  for  any  particular  item.  The  rating 
itself  would  not  reflect  that  judgment  process.  We  would  not  know 
from  the  rating  whether  the  evaluator's  YES  was  a  confident 
assessment  of  full  compliance  or  was  assigned  in  grudging 
acknowledgment  of  marginal  compliance. 

Considering  the  general  process  and  purpose  of  USI  design 
evaluation,  it  will  probably  prove  more  useful  if  the  evaluator's 
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judgment  is  reflected  more  directly  in  the  ratings  assigned.  The 
sample  checklist  application  in  Appendix  B  employs  numeric  ratings 
that  range  from  zero  (no  compliance)  to  ten  (full  compliance).  With 
such  numeric  ratings,  varying  degrees  of  marginal  compliance  can  be 
acknowledged  in  the  design  evaluation  record,  as  a  means  of  pointing 
the  way  toward  needed  design  improvements. 

Although  numeric  ratings  may  permit  an  evaluator  to  convey 
design  strengths  and  weaknesses  more  effectively,  we  must  recognize 
that  numeric  evaluation  can  be  misleading.  Qualitative  ratings  such 
as  YES/NO,  or  GOOD/FAIR/POOR,  are  clearly  judgmental.  Numeric 
ratings,  however,  may  imply  a  degree  of  objectivity  that  is  not 
justified  by  the  real  nature  of  the  design  evaluation  process.  Thus 
evaluators  must  be  conservative  when  reporting  and  Interpreting  the 
results  of  numeric  rating. 

As  a  practical  matter,  rating  compliance  with  agreed  design 
guidelines  will  involve  a  great  many  other  problems,  of  the  kind 
discussed  in  the  previous  section  of  this  report.  Particularly  in 
early  stages  of  system  development,  when  ratings  can  be  made  only  in 
terms  of  specifications  and  other  documentation  of  proposed  designs, 
it  may  not  be  possible  to  determine  the  degree  of  compliance  with 
various  specific  design  requirements.  For  some  items  in  the 
checklist,  compliance  ratings  may  have  to  be  marked  as  unknown. 

Even  for  a  fully  documented  and  completed  system,  it  is  obvious 
that  evaluation  of  compliance  with  design  objectives  must  be  in 
large  degree  judgmental.  And  again  we  must  consider  the  question  of 
whose  judgment  should  be  sought. 

Designers  may  wish  to  apply  the  checklist  to  their  own  work, 
perhaps  as  a  periodic  measure  to  determine  interim  success  in 
meeting  objectives.  Any  formal  evaluation,  however,  should  involve 
system  users  and  their  representatives.  They  may  choose  to  form 
their  judgments  in  consultation  with  designers,  but  they  should 
record  and  report  their  design  evaluation  independently,  in  their 
own  behalf. 


ANNOTATION 

Weightings  and  ratings  will  permit  a  quantitative  assessment  of 
USI  design,  but  in  many  cases  will  not  convey  the  exact  nature  of 
deficiencies  observed  during  design  evaluation.  It  will  help  in 
prescribing  improvements  to  USI  design,  if  the  evaluator  can  note 
specific  examples  of  design  deficiencies  for  low-rated  items  in  the 
checklist. 


Such  annotation  would  take  the  form  of  appended  comments. 

There  is  not  sufficient  space  in  the  checklist  itself  to  provide  a 
full  description  of  observed  deficiencies.  It  is  proposed  instead 
that  the  checklist  simply  include  for  any  item  an  indication  that  a 
comment  has  been  appended.  That  indication  might  be  a  check  or 
other  special  mark  following  the  assigned  rating.  Appended  comments 
would  be  labeled  with  the  number  codes  of  corresponding  checklist 
items . 

Evaluator  comments  will  not  always  describe  design 
deficiencies,  although  that  would  be  their  primary  function.  Some 
comments  might  raise  questions  about  design  features  considered 
doubtful  by  the  evaluator.  Other  comments  might  commend  observed 
examples  of  good  design  practice. 

Some  comments  might  pertain  to  the  weightings  assigned  to 
different  checklist  items,  dealing  with  design  requirements  rather 
than  design  achievement.  For  example,  there  might  be  comments 
explaining  why  particular  items  have  been  assigned  zero  weight. 
Although  such  comments  would  not  affect  design  evaluation,  they 
could  be  useful  if  design  requirements  must  later  be  reconsidered. 

Finally,  some  comments  might  pertain  to  functional  coverage, 
perhaps  indicating  a  special  interpretation  for  an  item  in  the 
checklist,  or  the  conversion  of  a  generally  stated  item  into  a  more 
specific  design  rule. 


PRACTICAL  APPLICATION 

Effective  application  of  the  design  evaluation  checklist  will 
require  that  user  interface  design  be  specified,  implemented,  and/or 
documented  in  sufficient  detail  to  permit  review  in  terms  of 
features  specified  in  the  checklist.  Otherwise,  many  of  the 
checklist  items  can  be  rated  only  with  question  marks.  In  such  a 
situation,  application  of  the  checklist  can  merely  indicate 
inadequacy  of  design  documentation  rather  them  inadequacy  of  the 
design  itself. 

In  its  initial  applications,  we  may  expect  that  the  design 
evaluation  checklist  will  be  used  solely  for  guidance  rather  than 
for  ccnuiactual  enforcement.  Certainly  if  a  contractor  has  not  been 
required  to  establish  and  follow  USI  design  guidelines  in  the  first 
place,  then  it  would  not  be  reasonable  to  impose  retroactively  some 
set  of  design  evaluation  criteria  derived  from  guidelines.  If  the 
design  evaluation  checklist  proves  useful  in  its  initial 
applications,  however,  we  may  see  its  future  adoption  as  a  more 
formal  contractual  instrument. 


SECTION  3 


POTENTIAL  COMPUTER  AIDS 


The  design  evaluation  checklist  proposed  here  should  prove  a 
useful  tool.  Looking  ahead,  however,  we  can  envision  ways  in  which 
this  tool  could  be  made  even  wore  effective.  A  promising  means  for 
facilitating  its  use  would  be  to  implement  the  checklist  as  an 
on-line  computer  tool.  Computer  aids  could  be  provided  for  making 
entries  to  the  checklist,  for  reviewing  and  tailoring  associated 
guidelines  material,  and  for  the  analysis  and  reporting  of  design 
evaluation.  These  topics  are  discussed  here. 


CHECKLIST  ENTRY  AND  REVIEW 

At  the  simplest  level,  a  checklist  maintained  on-line  in  a 
computer  could  help  the  entry  and  editing  of  evaluator  judgments  -- 
the  weightings  of  relative  item  importance,  the  ratings  of  judged 
design  compliance,  and  any  associated  annotation  including  evaluator 
comments . 

Checklist  entries  will  reflect  evaluator  judgment,  and 
judgments  may  change  from  time  to  time.  Changes  to  paper  forms  may 
be  messy  to  make,  and/or  involve  tedious  transcription  of  records. 

Moreover,  the  process  of  design  evaluation  will  often  be 
continuous,  repeated  at  different  stages  of  system  development. 
Successive  evaluations  will  generally  indicate  design  improvements. 
Ratings  for  many  items  will  stay  the  same,  but  ratings  for  some 
items  may  be  revised  upward,  and  further  comments  added  to  the 
evaluation  record.  Such  successive  changes  would  be  easier  to  make 
in  a  computer-stored  checklist  than  in  a  paper  record. 

Review  of  checklist  entries  would  also  be  easier  by  accessing 
computer-stored  records.  With  computer  aids,  an  evaluator  could 
request  direct  display  of  any  portion  of  the  checklist,  in  any  of 
its  successively  applied  versions,  without  having  to  leaf  through  a 
growing  sheaf  of  paper  records.  Computer  aids  would  be  especially 
helpful  for  displaying  checklist  annotation,  permitting  an  evaluator 
to  review  directly  the  comments  associated  with  any  selected  item, 
rather  than  having  to  search  separate  sheets  of  paper. 

Which  kind  of  checklist  will  prove  preferable  --  the  paper 
version  or  the  on-line  computer  aid  --  will  depend  upon  conditions 
of  use.  For  field  use,  where  an  evaluator  must  observe  a  system 
under  test,  the  paper  checklist  would  seem  more  convenient.  For 
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desk  use,  when  an  evaluator  is  reviewing  design  specifications, 
display  printouts,  etc.,  an  automated  checklist  could  of;er 
worthwhile  advantages.  That  automated  checklist  co»1d  fe  used  to 
help  transcribe  ratings  from  paper  checklist  records  tf<cen  in  the 
field. 


REVIEWING  GUIDELI  S  MATERIAL 

One  significant  feature  of  the  checklist  format  is  that  each 
item  expresses  a  design  guideline  in  condensed  wording,  with  no  full 
statement  of  the  guideline  nor  any  explanatory  examples  and  comment. 
An  evaluator  who  is  not  familiar  with  the  guidelines  material  will 
have  to  refer  back  and  forth  between  checklist  titles  and  their 
corresponding  source  guidelines.  In  effect,  an  evaluator  using  a 
paper  checklist  must  keep  at  hand  not  only  the  checklist  itself  but 
also  a  book  of  design  guidelines. 

A  better  alternative  would  be  to  reconfigure  the  paper 
checklist  to  become  an  on-line  checklist  supported  by  automated 
computer  aids,  including  computer  storage  and  display  of  associated 
guidelines  material.  Using  such  an  automated  checklist,  an 
evaluator  might  query  any  particular  item  to  request  immediate 
computer  display  of  explanatory  material  equivalent  to  that 
available  in  the  published  guidelines. 


TAILORING  GUIDELINES 

An  automated  checklist  could  be  of  even  greater  value  for 
design  evaluation  in  applications  where  guidelines  have  been 
tailored  to  become  system-specific  rules.  In  such  a  case,  published 
guidelines  would  not  provide  a  complete  account  of  the  desired 
design  features.  An  evaluator  would  have  to  refer  also  to  whatever 
additional  material  may  have  been  approved  for  design  guidance. 

Computer  aids  could  help  ensure  reliable  reference  to  agreed 
system-specific  design  rules,  in  effect  displaying  tailored  rather 
than  standard  guidelines  material.  With  such  an  automated 
checklist,  access  to  explanatory  material  would  certainly  be  easier. 
The  evaluator  would  not  need  a  book  of  design  guidelines,  plus 
whatever  further  notes  pertain  to  system-specific  tailoring. 

Instead,  the  evaluator  would  need  a  suitable  computer  terminal. 

Once  guidelines  had  been  tailored  with  computer  aids,  computer 
facilities  could  also  be  used  to  print  out  a  tailored  version  of  the 
design  evaluation  checklist.  That  revised  checklist  could  then  be 
used  in  field  testing  with  the  same  convenience  as  the  "standard" 
checklist  presented  here. 
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DESIGN  ANALYSIS 


Where  an  automated  checklist  could  really  help  an  evaluator  is 
in  analyzing  and  summarizing  the  ratings  that  have  been  made, 
particularly  if  differentia]  weightings  have  been  assigned  to  the 
various  items.  Computer  aids  for  analysis  could  automatically 
multiply  ratings  of  design  compliance  by  appropriate  weightings,  sum 
the  results  to  derive  an  overall  assessment  of  USI  design  quality, 
and  display  the  results  in  different  ways  to  provide  more  detailed 
diagnostic  information. 

Computer  analysis  could  provide  a  diagnostic  profile  assessing 
design  quality  for  the  different  functional  areas  supported  by  user 
interface  software,  for  identified  functions  within  those  areas,  and 
even  for  specific  highly-weighted  items  pertaining  to  particular 
functions.  A  sample  of  what  might  be  obtained  from  a  computer¬ 
generated  design  analysis  is  illustrated  in  Appendix  C.  Such 
diagnostic  evaluation  can  be  tedious  to  compute  manually,  but  would 
be  quite  easy  to  generate  with  appropriate  computer  aids. 


FUTURE  WORK 

For  use  at  ESD/MITRE,  a  logical  next  step  will  be  to 
reconfigure  the  checklist  as  an  automated  tool,  so  that  it  can  be 
supported  with  on-line  computer  aids  of  the  kind  suggested  here  — 
aids  for  explaining  checklist  items,  for  assigning  weightings  and 
ratings,  and  for  the  diagnostic  analysis  and  reporting  of  design 
evaluation  results.  An  on-line  interface  to  such  an  automated 
checklist  will  be  designed  in  FY85  under  MITRE  Project  5720.  Future 
implementation  and  testing  of  an  automated  design  evaluation 
checklist  will  require  further  work. 

Once  implemented,  an  automated  checklist  would  offer  potential 
benefits  beyond  its  immediate  use  for  USI  design  evaluation.  An 
automated  checklist  would  provide  an  example  of  how  computer  tools 
might  be  applied  more  generally  to  improve  corporate  effectiveness 
in  system  engineering  for  acquisition  programs.  We  can  anticipate 
that  such  computer  aids  for  focusing  expert  knowledge  in  support  of 
our  technical  work  will  contribute  strongly  to  Air  Force  system 
acquisition  in  the  decades  ahead. 
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APPENDIX  A 


USI  DESIGN  EVALUATION  CHECKLIST 


This  appendix  presents  the  checklist  for  evaluating  USI 
software  design.  As  discussed  in  the  text  of  this  report,  checklist 
coverage  is  governed  by  the  source  guidelines.  For  functions  such 
as  graphics  where  there  are  currently  few  guidelines,  or  none  at 
all,  checklist  coverage  is  correspondingly  weak. 

The  checklist  will  improve  over  the  next  several  years  as  the 
guidelines  themselves  improve.  Meanwhile,  means  might  be  found  to 
compensate  for  current  weaknesses  in  the  checklist,  perhaps  by 
expanding  coverage  in  weak  areas  to  add  specific  design  rules  for 
particular  system  applications. 

In  this  checklist,  the  item  descriptor  in  each  line  simply 
replicates  the  title  of  the  source  guideline.  Ideally,  all  of  those 
titles  should  be  informative,  adequate  in  themselves  to  convey  the 
intent  of  the  source  guidelines.  But  that  ideal  is  difficult  to 
achieve.  Users  ol  vMs  checklist  must  refer  to  source  guidelines  in 
ESP-TR-84-190  (Smith  '.nd  Hosier,  19S4b)  to  ensure  that  design 
requirements  are  clearly  understood  for  each  item. 

Within  the  checklist,  items  are  numbered  sequentially  for  each 
USI  function  in  order  to  permit  convenient  referencing.  Within  each 
function,  there  is  coverage  of  various  subordinate  topics  indicated 
by  the  item  titles.  Each  item  that  introduces  a  new  topic  is  marked 
with  a  black  dot  (•)  before  its  title.  When  succeeding  items  deal 
with  the  same  topic,  each  is  marked  with  a  white  dot  (o)  to  imply 
that  it  should  be  interpreted  in  conjunction  with  other  related 
items . 

In  practical  field  use,  the  design  evaluation  checklist  wou?  ’ 
differ  sc-^whut  ii*  fora.al  from  uhat  shown  nere.  There  should 
probably  be  a  cover  sheet  with  instructions  explaining  use  of  the 
checklist.  There  might  also  be  spare  sheets,  or  perhaps  blank  items 
inserted  in  the  checklist,  to  permit  recording  desired  USI  software 
features  that  are  not  represented  in  the  checklist,  and  spare  sheets 
for  appended  comments .  Those  are  omitted  here  in  the  interests  of 
compactness . 

Readers  who  wish  to  use  the  design  evaluation  checklist  are 
invited  to  contact  the  authors  of  this  report  in  order  to  exchange 
information  concerning  the  details  of  its  practical  application.  By 
sharing  our  experience  with  this  new  tool,  we  shall  all  learn  more 
quickly  how  it  may  best  be  used. 
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DESIGN  EVALUATION  FOR  USI  SOFTWARE 


System:  _ 

Date:  _ 

Evaluator:  _ 

Comment— | 
Rating— |  j 

Guideline—)  Weight—)  j  j 
Number  j  III 

V  V  V  V 


DATA  ENTRY _ General  |  1.0  | 

•  Entry  via  primary  display  -1 

°  Feedback  during  data  entry  -2 

o  Fast  response  -3 

•  Single  method  for  entering  data  -4 

•  Defined  display  areas  for  data  entry  -5 

•  Consistent  'icthod  for  data  change  -6 

•  User-paced  data  entry  -7 

•  Explicit  ENTER  action  -8 

o  ENTER  key  labeling  -9 

•  Explicit  CANCEL  action  -10 

•  Feedback  for  completion  of  data  entry  -11 

o  Feedback  for  repetitive  data  entries  -12 

o  Feedback  when  changing  data  -13 

•  Keeping  data  items  short  -14 

o  Partitioning  long  data  items  -15 

•  Optional  abbreviation  -16 

o  Distinctive  abbreviation  -17 

o  Consistent  abbreviation  rule  -18 

o  Minimal  exceptions  to  abbreviation  rule  -19 

o  Minimal  deviation  from  abbreviation  rule  -20 

o  Fixed  abbreviation  length  -21 

o  Clarifying  unrecognized  abbreviations  22 

•  Prompting  data  entry  -23 

•  Character  entry  via  single  keystroke  -24 

o  Minimal  shift  keying  -25 

•  Upper/ lower  case  equivalent  -26 

•  Decimal  point  optional  -27 

•  Leading  zeros  optional  -28 

•  Single/multiple  blanks  equivalent  -29 
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Comment— ] 
Rating— i  j 

Weight— i  |  | 


V  V  V 


DATA  ENTRY 


Position  Designation 


•  Distinctive  cursor  -1 

o  Non-obscuring  cursor  -2 

o  Precise  pointing  -3 

•  Explicit  activation  -4 

•  Fast  Response  -5 

•  Stable  cursor  -6 

•  Responsive  cursor  control  -7 

•  Consistent  incremental  positioning  -8 

o  Variable  step  size  -9 

o  Proportional  spacing  -10 

•  Continuous  cursor  positioning  -11 

•  Direct  pointing  -12 

o  Large  pointing  area  for  option  selection  -13 

•  Cursor  control  at  keyboard  -14 

•  Compatible  control  of  cursor  movement  -15 

•  Minimal  use  of  multiple  cursors  -16 

o  Distinctive  multiple  cursors  -17 

o  Distinctive  control  of  multiple  cursors  -18 

o  Compatible  control  of  multiple  cursors  -19 

•  Consistent  HOME  position  -20 

•  Consistent  cursor  placement  -21 

•  Easy  cursor  movement  to  data  fields  -22 

•  Display  format  protection  -23 


•  Data  entry  independent  of  cursor  placement  -24 


DATA  ENTRY 


Direction  Designation 


•  Analog  entry  of  estimated  direction  -1 

•  Keyed  entry  of  quantified  direction  -2 
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Comment—} 
Rating-}  j 
Weight-}  j  ! 


V  7  7 


DATA  ENTRY _ Text  I  1.3  | 


•  Adequate  display  capacity  -1 

•  Full  editing  capabilities  during  text  entry  -2 

•  Free  cursor  movement  -3 

o  Control  entries  distinct  from  text  -4 

•  Natural  units  of  text  -5 

o  Control  entry  based  on  units  of  text  -6 

o  Highlighting  specified  text  -7 

o  Cursor  movement  by  units  of  text  -8 

•  String  search  -9 

o  Upper/ lower  case  equivalent  in  search  -10 

o  Specifying  case  in  search  -11 

o  Global  search  and  replace  -12 

o  Case  in  global  search  and  replace  -13 

•  Automatic  pagination  aids  -14 

o  User  control  of  pagination  -15 

o  Controlling  integrity  of  text  unics  -16 

•  Automatic  line  break  -17 

o  Consistent  word  spacing  -18 

o  Hyphenation  by  users  -19 

•  Format  control  by  user  -20 

•  Establishing  predefined  formats  -21 

o  Storing  user-defined  formats  -22 

•  Moving  text  -23 

•  Storing  frequently  used  text  -24 

•  Necessary  data  displayed  -25 

o  Text  distinct  from  annotation  -26 

•  Printing  for  proofreading  -27 

•  Text  displayed  as  printed  -28 

•  Flexible  printing  options  -29 

•  Information  on  printing  status  -30 

•  Auditory  signals  for  alerting  users  -31 

•  Protecting  text  during  page  overruns  -32 

•  Confirming  actions  in  DELETE  mode  -33 

•  Reversible  actions  -34 

•  User  confirmation  of  editing  changes  -35 
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Comment—) 
Rating-)  j 
Weight-)  |  | 


7  V  V 


DATA  ENTRY  Data  Forms  B  1.4  B 

•  Single  entry  of  related  data  -1 

•  Flexible  irterrupt  -2 

•  Minimal  use  of  delimiters  -3 

o  Standard  delimiter  character  -4 

•  Data  field  labels  -5 

o  Consistent  labeling  -6 

o  Protected  labels  -7 

o  Labels  close  to  data  fields  -8 

•  Marking  field  boundaries  -9 

o  Prompting  field  length  -10 

o  Marking  required/optional  data  fields  -11 

•  Automatic  justification  of  variable- length 

entries  -12 

•  Explicit  tabbing  to  data  fields  -13 

•  Distinctive  label  format  -14 

o  Consistent  label  format  -15 

o  Label  punctuation  as  entry  cue  -16 

•  Informative  labels  -17 

•  Data  format  cueing  in  labels  -18 

o  Labeling  units  of  measurement  -19 

o  Familiar  units  of  measurement  -20 

o  Alternative  units  of  measurement  -21 

•  Form  compatible  for  data  entry  and  display  -22 

o  Form  compatible  with  source  documents  -23 

•  Minimal  Cursor  positioning  -24 

o  Data  items  in  logical  order  -25 

•  Automatic  cursor  placement  -26 


DATA  ENTRY _  Tables  I  1,5  I 

•  Tables  for  related  data  sets  -1 

•  Distinctive  labels  -2 

o  Informative  labels  -3 

•  Tabbing  within  rows  -4 

o  Tabbing  within  columns  -5 

•  Automatic  justification  of  entries  -6 

o  Justification  of  numeric  entries  -7 

•  Aiding  entry  of  duplicative  data  -8 

•  Row  scanning  cues  -9 


23 


Comment— i 
Rating— i  j 

Weight— i  j  | 

I  I  I 

7  V  V 


DATA  ENTRV _ Graphics  j  1.6  j 

(No  entries) 


DATA  ENTRY _ ' _ Data  Vcl  idation  I  1.?  J| 


•  Automatic  data  validation  -1 

•  Accepting  correct  entries  -2 

•  Non-disrupt ive  error  messages  -3 

•  Deferral  of  required  data  entry  -4 

o  Reminder  of  deferred  entry  -5 

•  Timely  validation  of  sequential  transactions  -6 

•  Optional  irem-by-item  validation  -7 


DATA  ENTRY _ Other  Data  Processing  1  1.8  1 


•  Default  values  -1 

o  User  definition  of  default  values  -2 

o  Display  of  default  values  -3 

o  Easy  confirmation  to  enter  default  values  -4 

o  Temporary  replacement  of  default  values  -5 

•  Automatic  generation  of  routine  data  -6 

•  Automatic  computation  of  derived  data  -7 

•  User  review  of  prior  entries  -3 

•  Automatic  entry  of  redundant  data  -9 

•  Automatic  cross-file  updating  -10 


DATA  ENTRY _ Design  Change  ■  1.9  | 

•  Flexible  design  for  data  entry  -1 
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Comment— i 
Rating—]  j 
Weight-]  !  j 

!  l  I 

V  V  V 


DATA  DISPLAY _ _  General  1  270 

•  Necessary  data  displayed  -1 

o  Only  necessary  data  displayed  -2 

•  Data  displayed  in  usable  form  -3 

•  Data  display  consistent  with  user  conventions  -4 

o  Establishing  display  standards  -5 

•  Consistent  display  format  -6 

•  User  control  of  data  display  -7 

o  User  changes  to  displayed  data  -8 

a  Protection  of  displayed  data  -9 

•  Context  for  displayed  data  -10 

•  Familiar  wording  -11 

o  Consistent  wording  -12 

o  Consistent  wording  across  displays  -13 

•  Minimal  use  of  abbreviation  -14 

o  Consistent  abbreviation  -15 

o  Distinctive  abbreviations  -16 

o  Dictionary  of  abbreviations  -17 

o  Minimal  punctuation  of  abbreviations  -18 


DATA  DISPLAY  _ "  ~Uata  Type  I  2.1  1 

•  Appropriate  data  types  -1 


25 


•  Conventional  text  display  "1 

•  Consistent  text  format  -2 
o  Conventional  use  of  mixed  case  -3 
o  Separation  of  paragraphs  “4 
o  Consistent  word  spacing  -5 

o  Minimal  hyphenation  -6 
o  Conventional  punctuation  -7 

•  Clarity  of  wording  “8 
o  Sentences  begin  with  main  topic  -9 
o  Simple  sentence  structure  -10 

o  Concise  wording  -11 
o  Distinct  wording  -12 
o  Affirmative  sentences  -13 
o  Active  voice  -14 
o  Temporal  sequence  -15 

o  Lists  for  related  items  -15 
o  Single-column  list  format  -17 
o  Logical  list  ordering  -18 
o  List  ordering  in  multiple  columns  -19 
o  Hierarchic  structure  for  long  lists  -20 


•  Abbreviations  defined  in  tej.t 


-21 


Comment— i 
Rating-!  i 
Weight-i  j  j 


V  7  V 


DATA  DISPLAY  -  Data  Type  Data  Forms  B  2.1.2  B 


•  Forms  for  related  data  -1 

•  Visually  distinctive  data  fields  -2 

o  Data  field  labeling  -3 

o  Descriptive  wording  of  labels  -4 

o  Consistent  wording  of  labels  -5 

o  Distinctive  wording  of  labels  -6 

o  Consistent  label  location  -? 

o  Distinctive  label  format  -8 

o  Labels  close  to  data  fields  -9 

o  Labeling  units  of  measurement  -10 

•  Consistent  format  across  displays  -11 

o  Form  compatible  for  data  entry  and  display  -12 

•  Consistent  format  within  data  fields  -13 

•  Partitioning  long  data  items  -14 

•  Distinguishing  blanks  from  nulls  -15 


DATA  DISPLAY  -  Data  Type  Tables  |  2.1.3  1 


•  Tables  for  data  comparison  -1 

•  Column  and  row  labels  -2 

o  Labeling  units  of  measurement  -3 

•  Justification  of  numeric  data  -4 

o  Justification  of  alphabetic  data  -5 

•  Logical  organization  -6 

o  Tables  referenced  by  first  column  -7 

o  Items  paired  for  direct  comparison  -8 

•  Distinctive  labeling  -9 

o  Numbered  items  start  with  "l"  -10 


o  Repeated  elements  in  hierarchic  numbering 

*  Row  scanning  cues 

o  Column  scanning  cues 
o  Consistent  column  spacing 

•  Consistent  label  format 


-11 

-12 

-13 

-14 

-15 
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Comment— i 
Rating— i  j 

Weight— i  |  j 


V  V  V 


DATA  DISPLAY  -  Data  Type _ Graphics  B  2.1.4  | 


•  Graphic  display  for  data  comparison  -1 

o  Graphic  displays  for  monitoring  data  change  -2 

•  Conventional  flowchart  orientation  -3 

•  Standarized  graphics  symbology  -4 


DATA  DISPLAY  -  Data  Type _ Combination  B  2.1.5  B 

•  Mixing  text  with  figures  -1 


DATA  DISPLAY _ Density  |  2.2  | 

•  Necessary  data  displayed  -1 

o  Only  necessary  data  displayed  -2 


DATA  DISPLAY 

Format  1  2.3  1 

•  Consistent  format 

-1 

o  Distinctive  display  elements 

-2 

•  Paging  crowded  displays 

-3 

o  Related  data  on  same  page 

-4 

o  Page  labeling 

-5 

•  Windows  for  related  data  sets 

-6 

o  Integrated  display 

-7 

o  Adequate  window  size 

-8 

•  Display  title  at  top 

-9 

o  Command  entry,  prompts,  messages 

at  bottom  -10 

•  Logical  data  oganization 

-11 

o  Grouping  for  data  comparison 

-12 

o  Data  grouped  by  sequence  of  use 

-13 

o  Data  grouped  by  function 

-14 

o  Data  grouped  by  importance 

-15 

o  Data  grouped  by  frequency 

-16 

o  Data  grouped  alphabetically  or 

chronologically 

-17 
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Comment—i 
Rating-i  | 
Weight-i  j  i 


V  7  V 


DATA  DISPLAY  Coding  I  2.4  | 

•  Highlighting  critical  data 

-1 

•  Coding  by  data  category 

-2 

•  Meaningful  codes 

-3 

o  Familiar  coding  conventions 

•4 

•  Definition  of  display  codes 

-5 

— 

— 

•  Consistent  coding  across  displays 

-6 

-  Alphanumeric  coding 

-7 

o  Consistent  case  in  alphabetic  coding 

-8 

o  Combining  letters  and  numbers 

-9 

o  Short  codes 

-10 

— 

— 

•  Special  symbols 

-11 

o  Consistent  use  of  special  symbols 

-12 

o  Markers  close  to  items  marked 

-13 

•  Shape  coding 

-14 

o  Establishing  standards  for  shape  coding 

-15 

— 

— 

•  Line  coding 

-16 

o  Underlining  for  emphasis 

-17 

o  Coding  by  line  length 

-18 

o  Coding  by  line  direction 

-19 

•  Size  coding 

-20 

— 

— 

o  Adequate  differences  in  size 

-21 

•  Brightness  coding 

-22 

o  Brightness  inversion 

-23 

•  Color  coding 

-24 

o  Conservative  use  of  color 

-25 

— 

— 

o  Adding  color  to  formatted  displays 

-26 

o  Redundant  color  coding 

-27 

o  Unique  assignment  of  color  codes 

-28 

o  Conventional  assignment  of  color  codes 

-29 

o  Limited  use  of  blue 

-30 

— 

— 

•  Blink  coding 

-31 

o  Blinking  marker  symbols 

-32 

o  Optimal  blink  rate 

-33 

•  Coding  with  texture,  focus,  motion 

-34 

•  Auditory  coding 

-35 

— 

— 

o  Distinctive  auditory  coding 

-36 

o  Voice  coding 

-37 

Comment—) 
Rating--)  I 
Weight—)  j  i 


V  V  V 


DATA  DISPLAY 


Generation  i  P  i 


•  User  selection  of  data  for  display  -1 

•  Display  identification  labels  -2 

o  Meaningful  display  labels  -3 

o  Consistent  format  for  display  labels  -4 

•  Fast  response  to  display  request  -5 

o  Signaling  completion  of  display  output  -6 

•  Regenerating  changed  data  -7 

o  Replacing  changed  data  -8 

•  Printing  displays  locally  -9 


DATA  DISPLAY  Framing  |  2.6  j 


•  Integrated  display  -1 

•  Easy  paging  -2 

o  Continuous  numbering  in  multipage  lists  -3 

o  Labels  for  multipage  tables  -4 

•  Annotating  display  of  continued  data  -5 

o  Numbering  display  pages  -6 

•  Consistent  orientation  for  display  framing  -7 

o  Windowing  with  free  cursor  movement  -8 

•  Labeling  display  framing  functions  -9 

o  Labeling  windowing  functions  -10 

o  Labeling  scrolling  functions  -11 


DATA  DISPLAY  Update  I  2,7  | 


•  Automatic  display  update  -1 

•  Readability  of  changing  data  -2 

o  Visual  integration  of  changing  graphics  -3 

•  Display  freeze  -4 

o  Labeling  display  freeze  -5 

o  Signaling  changes  to  frozen  data  -6 

o  Resuming  update  after  display  freeze  -7 


30 


C  ■BJLUJLW  A.'A'VJ'lllJ  FW ' T* 7^  IV ¥ 1  ff,  W*  3 ■  JL  ■ 


w wwrawwwf  ‘wuu  j vuwLwmjPHniiH'nwm wi 


Comment— i 
Rating-1  I 
Weight—)  |  | 

!  I  I 


DATA  DISPLAY _  Suppression  I  2.8  ■ 


•  Temporary  suppression  of  displayed  data  -1 

®  Labeling  display  suppression  -2 

•  Signaling  changes  to  suppressed  data  -3 

•  Resuming  display  of  suppressed  data  -4 


DATA  DISPLAY  Design  Change  I  2.9  | 

•  Flexible  design  for  data  display  -1 
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Comment—i 
Rating-i  ! 
Weight— i  j  | 
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SAMPLE  CHECKLIST  ENTRIES 


This  appendix  presents  a  portion  of  the  USI  design  evaluation 
checklist  showing  sample  entries  for  data  entry  functions.  In  the 
sample  shown  here,  numbers  have  been  entered  from  zero  to  ten  to 
represent  agreed  weightings  and  judged  ratings  of  different  design 
guidelines.  Those  numbers  are  fanciful,  and  are  included  here  only 
for  illustrative  purposes.  They  do  not  represent  a  design 
evaluation  of  any  real  system. 
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-24 

0 

o  Minimal  shift  keying 

-25 

0 

— 

•  Upper/ lower  case  equivalent 

-26 

0 

•  Decimal  point  optional 

-27 

0 

•  Leading  zeros  optional 

-28 

0 

•  Single/multiple  blanks  equivalent 

-29 

0 
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Comment— \ 
Rating— |  j 

Weight— (  j  | 


V  V  V 


DATA  ENTRY  Position  Designation 

i  i.i  i 

•  Distinctive  cursor 

-1 

10 

/£ 

o  Non-obscuring  cursor 

-2 

10 

Z 

o  Precise  pointing 

-3 

0 

• 

•  Explicit  activation 

-4 

10 

/£. 

. 

*  Fast  Response 

-5 

10 

r 

V' 

•  Stable  cursor 

-6 

3 

/* 

9  Responsive  cursor  control 

-7 

0 

•  Consistent  incremental  positioning 

-8 

10 

& 

o  Variable  step  size 

-9 

0 

o  Proportional  spacing 

-10 

0 

— 

•  Continuous  cursor  positioning 

-11 

0 

•  Direct  pointing 

-12 

10 

/_£ 

o  Lar£e  pointing  area  for  option  selection 

-13 

7 

£ 

•  Cursor  control  at  keyboard 

-14 

5 

/p 

Y' 

•  Compatible  control  of  cursor  movement 

-15 

0 

— 

•  Minimal  use  of  multiple  cursors 

-16 

0 

o  Distinctive  multiple  cursors 

-17 

0 

o  Distinctive  control  of  multiple  cursors 

-18 

0 

o  Compatible  control  of  multiple  cursors 

-19 

0 

•  Consistent  HOME  position 

-20 

3 

& 

•  Consistent  cursor  placement 

-21 

5 

4 

•  Easy  cursor  movement  to  data  fields 

-22 

3 

2 

^ ' 

•  Display  format  protection 

-23 

7 

• 

•  Data  entry  independent  of  cursor  placement  -24 

7 

DATA  ENTRY  Direction  Designation 

TT.2I 

•  Analog  entry  of  estimated  direction 

-1 

0 

•  Keyed  entry  of  quantified  direction 

-2 

0 

• 
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Comment— | 
Rating—)  f 
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1 

V 

V 

DATA  ENTRY  Text  R  1 

73  | 

•  Adequate  display  capacity 

-1 

10 

/£ 

“  Full  editing  capabilities  during  text  entry 

-2 

5 

3 

*  Free  cursor  movement 

-3 

5 

/£. 

V 

o  Control  entries  dlotinct  from  text 

-4 

5 

'/£ 

. 

«  Natural  units  of  text 

-5 

8 

X 

V 

o  Control  entry  based  on  units  of  text 

-6 

3 

c  Highlighting  specified  text 

-7 

3 

- 

o  Cursor  movement  by  units  of  text 

-8 

r% 

o 

£ 

ir' 

•  String  search 

-9 

c 

o  Upper/ lower  case  equivalent  in  search 

-10 

0 

— 

o  Specifying  case  in  search 

-11 

0 

o  Global  search  and  replace 

-12 

0 

o  Case  in  global  search  and  replace 

-13 

0 

»  Automatic  pagination  aids 

-14 

0 

o  User  control  of  pagination 

-15 

0 

— 

o  Controlling  integrity  of  text  units 

-16 

0 

•  Automatic  line  break 

-  i  7 

8 

/Z 

c  Consisten:  word  spacing 

-18 

8 

/£. 

o  Hyohenation  by  sers 

-19 

5 

•  Format  control  by  user 

-20 

10 

z 

•  Establishing  predefined  formats 

-21 

10 

o  Storing  user-defined  formats 

-22 

5 

'  7 

*  Moving  text 

-23 

1 

A 

V 

*  Storing  frequently  used  text 

-24 

10 

• 

•  Nc(  essary  data  displayed 

-25 

10 

Z 

V 

e  Text  distinct  from  annotation 

-26 

5 

£ 

•  Printing  for  proofreading 

-27 

0 

•  Text  displayed  as  printed 

-28 

0 

*  Flexible  printing  options 

-29 

0 

•  Information  on  printing  status 

-30 

0 

— 

*  auditory  signals  for  alerting  users 

-31 

8 

Jl 

l 

•  Protecting  text  during  page  overruns 

-32 

5 

/a 

• 

•  Confirming  actions  in  DELETE  mode 

-33 

10 

. 

•  Reversible  actions 

-34 

6 

/A 

A 

•  User  confirmation  of  editing  changes 

-35 

3 

A 

• 

52 


Comment— i 
Rating-i  I 
Weight— i  |  | 

I  I  I 

7  7V 


DATA  ENTRY  Date  Forms  1  1.4  | 

•  Single  entry  of  related  data 

-1 

10 

/e 

•  Flexible  interrupt 

-2 

8 

/* 

* 

•  Minimal  use  of  delimiters 

-3 

0 

o  Standard  delimiter  character 

-4 

0 

* 

•  Data  field  labels 

-5 

10 

o  Consistent  labeling 

-6 

10 

L 

V 

o  Protected  labels 

-7 

7 

/0 

# 

o  Labels  close  to  data  fields 

-6 

3 

‘/D 

# 

•  Marking  field  boundaries 

-9 

6 

X 

o  Prompting  field  length 

-10 

3 

/£ 

• 

o  Marking  required/optional  data  fields 

-11 

0 

•  Automatic  justification  of  variable- length 

entries 

-12 

0 

# 

•  Explicit  tabbing  to  data  fields 

-13 

8 

• 

•  Distinctive  label  format 

-14 

9 

'  2 

lr' 

o  Consistent  label  format 

-15 

7 

x 

If 

o  Label  punctuation  as  entry  cue 

‘•16 

0 

. 

•  Informative  labels 

-17 

6 

z 

•  Data  format  cueing  in  labels 

-18 

0 

* 

o  Labeling  units  of  measurement 

-19 

0 

o  Familiar  units  of  measurement 

-20 

0 

— 

* 

o  Alternative  units  of  measurement 

-21 

0 

♦ 

•  Form  compatible  for  data  entry  and  display 

-22 

10 

. 

o  Form  compatible  with  source  documents 

-23 

0 

« 

•  Minimal  Cursor  positioning 

-24 

6 

JL 

• 

o  Data  items  in  logical  order 

-25 

10 

X 

lr 

•  Automatic  cursor  placement 

-26 

3 

X 

DATA  ENTRY  Tables  N 

1.5  | 

•  Tables  for  related  data  sets 

-1 

10 

/p 

•  Distinctive  labels 

-2 

2 

10 

o  Informative  labels 

-3 

4 

fO 

•  Tabbing  within  rows 

-4 

10 

o  Tabbing  within  columns 

-5 

10 

•  Automatic  justification  of  entries 

-6 

0 

o  Justification  of  numeric  entries 

•7 

0 

•  Aiding  entry  of  duplicative  data 

-8 

0 

•  Row  scanning  cues 

-9 

G 

/J2. 

/ 
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Comment- i 
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DATA  ENTRY 


Graphics  |  1.6  j 


(No  entries) 


SATA  ENTRY _ '  Data  Validation  j  1.7  | 


1  Automatic  data  validation  -1 

•  Accepting  correct  entries  -2 

•  Non-disruptive  error  messages  -3 

•  Deferral  of  required  data  entry  -4 

o  Reminder  of  deferred  entry  -5 


10  /o_  . 

10  4 

4  ,t  . 
0  '  _  ^ 
0 


•  Timely  validation  of  sequential  transactions  -6  0 

•  Optional  item-by-item  validation  -7  0 


DATA  ENTRY _ Other  Data  Processing  B  1-8  j 


•  Default  values 

o  User  definition  of  default  values 
e  Display  of  default  values 
o  Easy  confirmation  to  enter  default  values 
o  Temporary  replacement  of  default  values 

•  Automatic  generation  of  routine  data 

•  Automatic  computation  of  derivea  data 

•  User  review  of  prior  entries 

•  Automatic  entry  of  redundant  data 

•  Automatic  cross- file  updating 


-1 

A 

1> 

-2 

0 

-3 

0 

-4 

0 

-5 

0 

— 

-6 

10 

-7 

10 

X 

-8 

0 

-9 

0 

-10 

10 

& 

DATA  ENTRY 


Design  Change  j  1.9 


10  J  v 


Flexible  design  for  dati  entry 


APPENDIX  C 


SAMPLE  DESIGN  ANALYSIS 


The  following  pages  illustrate  the  kind  of  analysis  that  could 
be  based  on  the  USI  design  evaluation  checklist.  The  example  shown 
here  assumes  that  the  sample  checklist  for  data  entry  functions 
invented  in  Appendix  B  has  been  extended  to  include  all  USI 
functional  areas.  Since  that  checklist  represents  a  fictional 
evaluation,  this  design  analysis  is  also  fictional,  and  does  not 
represent  an  evaluation  of  any  actual  system. 

In  this  sample  analysis,  it  is  assumed  that  an  evaluator  has 
assigned  ratings  to  all  required  items  in  the  checklist.  This  is 
indicated  in  the  computer-generated  report.  In  an  actual  system 
evaluation,  that  will  not  always  be  true.  Some  items  may  have  been 
left  unrated  because  of  inadequate  design  information.  In  such  a 
case,  the  computer  might  provide  a  partial  design  analysis.  Or,  if 
many  ratings  have  been  omitted,  the  computer  might  report  that  no 
useful  analysis  can  be  made. 

The  analysis  begins  at  a  general  level,  assessing  overall  USI 
design  quality  in  relation  to  defined  requirements.  The  analysis 
continues  at  levels  of  increasing  detail,  to  provide  a  diagnostic 
evaluation  of  design  compliance  for  different  USI  functional  arses, 
and  for  specific  functions  within  each  area,. 

Various  information  is  provided  at  different  levels  of 
analysis,  including  derived  measures  of  design  quality.  The 
measures  proposed  here  illustrate  several  promising  possibilities. 
Still  other  measures  night  be  derived  from  the  itemized  design 
evaluation  checklist  to  match  particular  system  requirements. 

As  noted  in  the  text  of  this  report,  such  a  detailea  design 
analysis  can  be  tedious  using  manual  methods,  although  it  is 
certainly  feasible.  It  would  be  helpful  to  provide  computer  aids 
for  such  an  analysis.  If  the  weightings  and  ratings  of  USI  design 
evaluation  can  be  assigned  on-line,  as  entries  to  an  automated 
checklist,  then  it  should  be  possible  to  provide  a  computer¬ 
generated  summary  and  detailed  diagnostic  information,  quickly  and 
accurately.  The  computer  output  might  look  much  like  that  imagined 
in  the  following  pages. 
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DESIGN  EVALUATION  FOR  USI  SOFTWARE 


System: 

Date: 

Evaluator: 


COMCON  (proposed  design) 
15  August  1984 
S.  L.  Smith 


SUMMARY 


All  required  items  have  been  rated,  and  so  a  comprehensive  analysis 
of  user  interface  software  can  be  made. 

Overall  achievement  of  USI  design  objectives  is  here  estimated  at 
|  78  |  percent.  This  figure  represents  a  weighted  average  of 
assigned  ratings  for  all  required  items,  in  relation  to  maximum 
possible  ratings. 

This  figure  indicates  generally  good  achievement  of  design 
objectives,  but  with  a  need  for  further  improvement.  A  profile  of 
weighted  design  quality  in  six  USI  functional  areas  shows  that 
improvement  is  needed  in  the  design  of  user  guidance  and  data 
protection  functions: 


Data  Entry 
Data  Display 
Sequence  Control 
User  Guidance 
Data  Transmission 
Data  Protection 


Design  Quality  (Percent) 

78%  overall 

V 


A 


A  more  detailed  evaluation  of  USI  design  is  presented  in  the  following 


FUNCTIONAL  AREAS 


Mean  weights  and  ratings  of  design  features  for  six  USI  functional 
areas  are  tabulated  below. 


(A) 


(B)  (C)  (D) 


(E) 


(F)  (G) 


Design 

Guidelines 

Listed  Required 
No.  % 


1. 

Data  Entry 

"1 - 

|  143 

83 

58 

T~ 

1 

| 

6.9 

8.3 

~ T~ 

1 

| 

4846 

- 1 

85  | 

2. 

Data  Display 

1 

|  163 

138 

85 

1 

1 

I 

7.8 

7.9 

1 

8678 

81  | 

3. 

Sequence  Control 

!  168 

1 

49 

29 

1 

1 

I 

7.3 

8.6 

1 

1 

| 

3010 

84  | 

4. 

User  Guidance 

1  97 

39 

40 

1 

1 

8.1 

6.4 

1 

1 

1 

1977 

62  | 

5. 

Data  Transmission 

|  40 

16 

40 

1 

1 

1 

7.7 

8.9 

1 

| 

1145 

93  | 

6. 

Data  Protection 

!  68 

J _ 

56 

82 

1 

6.3 

6.9 

1 

1 

— U 

2358 

66  | 
- 1 

Required 

Guidelines 

Mean  Mean 
Weight  Rating 


Computed 

Design 

Quality 

Value  % 


Overall 


679 


381  56 


7.3 


7.8 


22014  78 


(A)  =  total  number  of  guidelines  in  the  design  evaluation  checklist. 

(B)  =  number  of  required  guidelines,  i.e.,  those  weighted  above  zero. 

(C)  =  percent  of  listed  guidelines  that  are  required.  [=  100  x  B/A] 

(D)  =  mean  weight  (1-10)  assigned  to  required  guidelines. 

(E)  =  mean  rating  (0-10)  of  design  compliance  for  required  guidelines. 

(F)  =  computed  sum  of  weighted  design  compliance,  i.e.,  total  of 

weights  multiplied  by  ratings  for  required  guidelines. 

(G)  =  percent  of  actual  design  quality  achieved  in  relation  to  maximum 

potential  compliance. 

[-  100  x  F/(BxDxl0)  subject  to  rounding  errors] 
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Overall  design  bias  computed  from  this  table  is  1.5  percent.  Design 
bias  refers  to  a  tendency  for  compliance  to  be  greater  for  required 
design  features  that  are  assigned  greater  weight.  Design  bias  is 
computed  as  100  x  (F/(BxDxE)  -  1). 

The  computed  bias  indicates  that  design  compliance  for  this  system  is 
not  significantly  related  to  the  weighting  of  required  design 
features.  For  further  improvement  of  USI  design,  it  will  be  necessary 
to  examine  instances  in  which  high-weighted  features  have  received  low 
ratings  of  design  compliance. 

A  more  detailed  evaluation  is  presented  separately  for  each  USI 
functional  area  in  the  following  pages. 


DATA  ENTRY 


Mean  weights  and  ratings  of  design  guidelines  for  data  entry  functions 
are  tabulated  below.  For  a  more  detailed  review  of  data  entry 
functions,  consult  items  in  that  section  of  the  dSI  design  evaluation 
checklist. 


(A)  (B) 

(C) 

(D)  (E) 

(F)  (G) 

t 

|  Design 

Required 

1 

Computed  | 

|  Guidelines 

Guidelines 

Design  1 

1 

Quality 

1  Listed  Required 

Mean  Mean 

|  No. 

% 

Weight  Rating 

Value  %  1 

1 . 0  General 

i - 

|  29 

17 

59 

-h 

1 

5.6 

8.2 

-1 - 

|  814 

87 

■i 

1.1  Position 

1  24 

14 

58 

1 

6.8 

8.9 

j  882 

88 

1.2  Direction 

i  2 

0 

0 

- 

- 

j 

- 

1.3  Text 

j  35 

23 

66 

I 

6.6 

7.3 

I  1211 

80 

1.4  Data  Forms 

1  26 

16 

62 

1 

7.3 

8.4 

|  969 

84 

1.5  Tables 

1  9 

6 

67 

1 

7.3 

10.0 

1.6  Graphics 

1  o 

- 

- 

1 

- 

- 

1  - 

- 

1.7  Data  Validation 

j  7 

3 

43 

1 

8.0 

9.7 

I  230 

96 

1.8  Other  Processing 

1  io 

3 

30 

n 

9.3 

j  280 

93 

1 . 9  Des ign  Change 

|  1 

1 

1 

100 

H 

2.0 

mm 

20 

j 

Overall  Data  Entry 

1 

|  143 

J _ 

83 

58 

1 

6.9 

8.3 

i 

|  4846 

J _ 

85 

n 

1 

J 

Evaluator  comments  on  specific  guidelines  are  provided  for  31  items. 


The  following  high-weighted  guidelines  with  low  ratings  of  design 
compliance  will  require  special  attention  to  improve  USI  design. 


1.1-5 

Fast  response 

Weight 

10 

1.4-5 

Data  field  labels 

10 

1.9-1 

Flexible  design  for  data  entry 

10 

Rating 


5 

2 

2 


[A  complete  design  analysis  would  continue  with  similar  pages  for  five 
other  USI  functional  areas.  Those  pages  are  omitted  in  this  sample.] 


